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ABSTRACT 



The effects of academic self-concept and the learning 
environment on science and mathematics achievement were studied in remote and 
rural Western Australia. The Western Australian School Effectiveness Study 
explored achievement in 28 urban and rural high schools. In the first phase, 
survey instruments were developed and piloted in two schools, and in the 
second phase, a 3 -year longitudinal study of government, Catholic, and 
independent secondary schools was implemented. The proposed third phase will 
use a case study approach to examine some exceptionally effective schools. 

All results tended to be lower for students from remote areas, but it was 
suspected that these variations may be related to socioeconomic status. 
Teachers perceived that students were more supportive in metropolitan 
schools, and students in rural schools thought that their teachers and peers 
were more supportive. Students in country schools appeared to be more 
satisfied with their schools. Achievement scores were not readily comparable 
because of a lack of prior achievement data. Background and context effects 
explained a large amount of variance in the classroom, but little was found 
to explain student achievement at the school or classroom level of analysis. 
Most variation in science and mathematics achievement was at the student 
level. (Contains 4 figures, 15 tables, and 106 references.) ( SLD) 
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A MULTILEVEL ANALYSIS OF SCIENCE AND MATHEMATICS ACHIEVEMENT 



Deidra J. Young 
Australian Research Fellow 
Science and Mathematics Education Centre 
Curtin University of Technology 
Perth, Western Australia 



INTRODUCTION 

The purpose of this study was to investigate the effect of academic self-concept and the learning 
environment on science and mathematics achievement among students in rural and remote parts of 
Western Australia. This study was prompted by some of the findings from the Tomlinson Review of 
Rural Schools (Tomlinson, 1994). Further comparisons of rural, remote and urban schools revealed 
the significant effect which socioeconomic status has upon student outcomes (Young, 1991a, 1991b, 
1994a, 1994b, 1994c, 1994d, 1994e, 1995; Young & Fraser, 1992a, 1992b, 1993a, 1993b, 1994). 

This paper reports findings from a study undertaken in 28 urban and rural schools in Western 
Australia, called the Western Australian School Effectiveness Study (WASES). While these schools 
were from entirely different locations, they provided interesting information about how students and 
teachers perceived their schools. 

REVIEW OF RURAL EDUCATION RESEARCH 

Recent educational research has examined rural/urban differences in achievement, appropriateness of 
rural/urban achievement measures, effects of parents and the community on the attainment of rural 
students, and how well rural students succeed in higher education. However, in order to assess the 
rural school's impact on student outcomes, rural/urban comparisons must be made on students who 
are matched by background and school characteristics. Findings that there are little differences in the 
academic achievement of rural and urban students or in their desire to attend higher education and that 
rural students aspire to higher education contrast with evidence that rural high school students have 
less total access to educational information. It could be argued that rural high school students are 
therefore, in terms of their overall progress, achieving more, not less, in spite of greater obstacles 
(Edington & Koehler, 1987). 

Many educators, researchers, legislators, and the general public believe that students from smaller and 
rural schools receive an education that is inferior to that of students from larger urban or suburban 
schools. Until recently, there has been little empirical evidence to challenge that view. Now, however, 
a growing body of work has begun to examine how well students perform in and after graduation 
from rural schools. Some of these studies are presented below, and, although the results are far from 
conclusive, they do suggest that some generally held beliefs about rural student achievement are open 
to question. 

A comparison of the performance on standardised achievement tests of students from small, usually 
rural, schools with those from larger, often urban, institutions has not produced definitive results. 
Several studies have not found any significant differences between the two groups. In research 
completed in the state of New York, Monk and Haller (1986) found that students from smaller (often 
rural) schools achieved as well as students from larger schools. Kleinfeld and others (1985), in their 
Alaska study, did not find that high school size determined the quality of a student's education, 
experience, or achievement on standardised tests. Moreover, in one New Mexico study, which looked 
at factors affecting performance of selected high school students, those attending schools in rural areas 
performed as well as those in urban locales (Ward & Murray, 1985). 

There is some indication that what is being measured in studies of rural-urban differences is 
socioeconomic status and/or ethnicity. Easton and Ellerbruch (1985) found that the poorer rural 
students scored considerably lower on citizenship and social studies tests than did students from upper 
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socioeconomic urban communities. Another study which held socioeconomic level and ethnicity 
constant revealed no urban-rural achievement gap (Edington & Martellaro, 1984). This has been 
reaffirmed by an Australian study of students in Years 3, 7 and 10. In this study, the socioeconomic 
status of the school accounted for most of the variation in student achievement in mathematics, reading 
and writing (Young, 1994b, 1994d). However, this research was limited in that the variables were 
collated on a post hoc basis. 

Kleinfeld and colleagues (1985) suggested that schools that achieve the best results do exhibit strong 
teacher/administration/community partnership and school/community agreement on educational 
programs. They also have reported that there is a direct relationship between quality education 
programs and the ability of the staff to work toward an educational partnership with the community. 
Smaller communities tend to generate more community support for the school, with the school 
becoming a centre for community activity. This, in turn, theoretically provides the students with a 
greater feeling of belonging to something in which they can participate, and thus enables them to 
develop a better self-concept. 

Rural education research has been conducted in Louisiana by Stringfield and Teddlie (1991, 1993) for 
ten years and these researchers have produced some fine and valuable findings. These researchers 
have found significant variations in what makes a school effective in the rural parts of the USA. 
Further research by Bobbett and colleagues (1990) in the rural Appalachian regions of Kentucky and 
Tennessee identified effective schools were characterised by school climate factors including: 

• orderly and pleasant 

• characterised by purposeful activity 

• caring 

• concerned about student and community involvement 

• celebrating achievements of students and faculty 

• staffed with highly committed individuals 

More importandy, researchers have now begun to question the generalisability of the effective school 
model for urban, suburban and rural schools, given substantial differences in their social and 
organisational environments (Hannaway & Talbert, 1993). Rural educational research must therefore 
not continue to ignore contextual effects and this study presents findings which have accounted not 
only for the rural sector, but also school context and student background. 

REVIEW OF SCHOOL EFFECTIVENESS RESEARCH 

In early research on school effectiveness, there was considerable emphasis on the ability and famil y 
background of the student in determining academic performance. The Coleman Report (Coleman et 
al., 1966, p. 296) estimated that the school influence on student achievement was about 10 to 20 
percent of the total variance, yet the methodology used by Coleman had not accounted for the 
hierarchical nature of students nested within schools. Coleman’s findings were repeated in further 
large-scale studies (Jencks et al., 1972, 1979; Hauser, Sewell & Alwin, 1976), which suggested that 
(1) school level variables, such as physical resources, account for small amounts of variability in 
student achievement and (2) student characteristics, such as socioeconomic status and home 
background, should be used to adjust student achievement in statistical analysis of large-scale studies. 

In Britain, research into schools became prominent during the 1980s with Fogelman’s findings that the 
amount of schooling received by students was directly related to their academic achievement (1978, 
1983). While early British researchers analysed the effects of academic and social backgrounds of 
students, there was some doubt about whether control for differences in student intake was a dequate 
(Reynolds, 1976; Reynolds & Sullivan, 1979; Rutter et al., 1979). Reynolds reported large school 
level differences in attendance rates, even when students came from similar social and economic 
backgrounds. More recent studies, which included student information prior to school entry and better 
analytic techniques, reported substantial variations between schools (Mortimore et al., 1988; Smith & 
Tomlinson, 1989; Nuttall et al., 1989). The improvement of analytical techniques more adequately 
addressed the hierarchical nature of the data, that is, the variability between schools and within schools 
was separated (Bryk & Raudenbush, 1986; Goldstein, 1984, 1987). 

While early British research by Reynolds (1982) and Rutter and colleagues (1979) indicated that 
schools affected students equally, later studies by Aitkin and Longford (1986) found significant 
differences in school effects for students from different socioeconomic backgrounds Further 
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78-79) reported that achievement was significantly greater for students from more 
affluent home backgrounds, when compared with students from poorer homes. In this British study, 

sch0 4 ° mtak u- dlfferences account for a large proportion of the variation in 
unadjusted variation in student achievement Finally, Cuttance asserted that any analyses of the 
effectiveness of schools need to adjust for the social background and prior attainment of students. 

T^vinfnQQOw 11 ofsocial gender differences in United States schools has led researchers such as 
Levine (1992) to recommend that multiple measures of students’ social and economic background be 
used to control for social class influences on achievement. Levine et al. (1979) found that the 
frequently used US indicator, students’ subsidised lunch status, was not useful due to highly variable 
reporting by principals. Levine also urged that schools be examined for their effectiveness in 
equalising the academic achievement of minorities and disadvantaged groups. The importance of 
examining the equity of the school, as well as the school’s effectiveness, was advocated by US 
researchers who found that a school could be identified as highly effective, yet have lower class and 
minority students with poor academic performance (Brookover, 1985; Shoemaker, 1984; Lezotte, 



. im P ort anceof the school ; and classroom environment in enhancing learning has been investigated 
py Eraser (1986, 1991) who found strong links between student outcomes and their educational 
environments. Fraser and Tobin combined qualitative and quantitative methodologies in their study of 
exemplary teachers and found the classroom learning environment was decisive in enhancing student 
learning in science (Fraser & Tobin, 1989; Tobin & Fraser, 1987). In addition, studies into factors 
associated with educational productivity found nine consistent factors: student ability, student 
development, student motivation, instructional time, instructional quality, home environment 
classroom environment, peer groups and television viewing (Fraser, Walberg, Welch & Hattie, 1987)! 

CONTEXT STUDY IN SCHOOL EFFECTIVENESS RESEARCH 



Educational researchers who were challenged by the Coleman report’s conclusion that schools don’t 
matter, set about to mvestigate schools that served low-SES students who performed well on 

( S r ?. 0k ,°n V - 7 e I ?, aL ’ , 1979; Brookover & Lezotte, 1979; Edmonds, 1979b; Glenn, 
’ ^74; Venezky & Winfield, 1979). As the research developed on effective 
schools for the urban poor”, Edmonds repudiated Coleman and Jencks: 



Repudiation of the social science notion that family background is the 
principal cause of pupil acquisition of basic skills is probably 
prerequisite to successful reform of public schools. 

(Edmonds, 1979a, p. 23) 



However, in a new phase of school effects research, sampling procedures were improved and the 
characteristics of effective schooling for students from a variety of contexts were questioned 
(Wimpelberg, Teddlie & Stringfield, 1989). These researchers began to ask what makes an effective 
school for the disadvantaged groups in our community. Further, Levine and Lezotte (1990) concluded 
that three types of school contexts should be studied in school effects research- 



• Student body SES (socioeconomic status of the aggregated student population) 

• Grade level of schooling 

• Urbanity (rural versus urban) 

Stringfield and Teddlie’s ten year longitudinal study of effective schools in Louisiana cumulated in 
significant findings that there were six types of differentially effective schools (1993). These 
researchers found that students in more effective schools had higher future educational expectations 
than those from less effective schools. For these students, they felt less academic futility and 
perceived greater teacher push than did those students from less effective schools. There was a more 
positive educational climate for students from more effective schools. 



While some effective-school characteristics were found regardless of the school SES, such as clear 
academic mission, orderly environment, high academic engagement and frequent monitoring of 
stl ?jj, nt s P[Og res s, there were a number of differences in characteristics of effectiveness between 
middle- and low-SES schools (p. 36). A difference in future educational expectations by teachers in 
two types of schools was associated with effectiveness. They found that teachers in effective low- 
SES schools held high present, but more modest future, expectations for their students. 
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Of significant importance was the differences in effectiveness of schools depending upon the urbanity 
context of the school. Stringfield and Teddlie summarised 16 characteristics of differentiation between 
urban, suburban and rural elementary schools (1993, pp. 158-162). For example, ‘in small towns, an 
effective rural principal can help the school to become the focal point of the community and gamer 
additional resources along the way’. 

CONCEPTUAL FRAMEWORK 

A conceptual framework for school effectiveness described by Scheerens and Creemers (1989) in 
Figure 1. Scheerens and Creemers proposed a multi-level model of schooling which incorporates three 
organisational levels: the student, the classroom and the school (Creemers & Scheerens, 1994). 
These three levels were investigated using multilevel modelling of a large-scale, longitudinal survey 
data as suggested Stringfield and Teddlie’s contextually sensitive research studies (1993). Three types 
of contexts will be included in this research: student aggregated SES, grade-level and urbanity. 




Figure 1. Multilevel Model of School Effectiveness 
(Scheerens and Creemers, 1989) 



RESEARCH DESIGN 

This research study, the Western Australian School Effectiveness Study [WASES] involves three 
phases (Table 1). In the First Phase , the survey instruments were developed and piloted in two 
schools (1995). 

In the Second Phase , a three year longitudinal survey was commenced in 28 West Australian high 
schools. Both Government, Catholic and Independent secondary schools were surveyed. The 
purpose of this survey was to evaluate the school and classroom climate and characteristics of effective 
schools in differential contexts. Because the growth model is particularly useful for measuring change 
over time in student outcomes, while controlling for other influencing variables which may also 
change over time, the same students at the same schools will be surveyed over a period of three years 
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(1996 to 1998). This phase will be called WASES-H in 1996, WASES-IH in 1997 and WASES-IV in 
1998 and is being funded in part by the Australian Research Council. 

Finally, in the Third Phase, a case study approach will be used to examine some exceptionally 
effective schools in the rural and urban locations of Western Australia (1997-99). Case studies will 
commence in 1997 and selected from some outlier schools based on statistical data from WASES-H 
and WASES-m. 



Table 1. Longitudinal sampling frame. 



Phases of Study 


Year 


Grade of Sample 


Schools 


Students 


Phase 1 
[WASES-I] 


1995 


Year 10 


2 Secondary Schools 


163 


Phase 2 

[WASES-H] 


1996 


Years 8, 9, 10 


28 Secondary Schools 


3,500 


[WASES-HI] 


1997 


Years 9, 10, 11 


28 Secondary Schools 




[WASES-IV] 


1998 


Years 10, 11, 12 


28 Secondary Schools 




Phase 3 
[WASES-V] 


1997-99 


Case Studies 


8 Rural and Urban Schools 








- Effective and Ineffective 



Assessing the school environment 

International research efforts involving the conceptualisation, assessment and investigation of 
perceptions of psychosocial aspects of educational environments have established educational 
environment as an important field of study (Fraser, 1994; Fraser & Walberg, 1991). One of the 
originators of this line of research. Moos (1974), found that the same three general categories can be 
used in conceptualising the individual dimensions characterising diverse psychosocial environments. 
This finding emerged from Moos's work in a variety of environments including hospital wards, 
school classrooms, prisons, military companies, university residences and work milieus. The three 
basic types of dimensions are: Relationship Dimensions (e.g., peer support, involvement) which 
identify the nature and intensity of personal relationships within the environment, and assess the extent 
to which people are involved in the environment and the extent to which they support and help each 
other; Personal Development Dimensions (e.g., professional interest) which assesses the basic 
directions along which personal growth and self-enhancement tend to occur; and System Maintenance 
and System Change Dimensions (e.g., innovation, work pressure) which involve the extent to which 
the environment is orderly, clear in expectations, maintains control and is responsive to change. 

Recent classroom environment research has focused on science laboratory classroom environments 
(McRobbie & Fraser, 1993), constructivist classroom environments (Taylor, Dawson & Fraser, 1995) 
and computer-assisted instruction classrooms (Teh & Fraser, 1995), while other studies have focused 
on the school environment (Fisher, Fraser & Wubbels, 1993). However, a careful review of the 
potential strengths and problems associated with existing school environment instruments led to the 
development of a new school environment instrument named the School Level Environment 
Questionnaire (SLEQ) (Fisher & Fraser, 1990), which measures teachers' perceptions of psychosocial 
dimensions of die school environment. This instrument consists of seven scales, with two measuring 
Relationship Dimensions (Student Support, Affiliation), one measuring the Personal Development 
Dimension (Professional Interest) and five measuring System Maintenance and System Change 
Dimensions (Staff Freedom, Participatory Decision Making, Innovation, Resource Adequacy and 
Work Pressure). 
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Table 2. Description of scales in SLEQ and their classification according to Moos' scheme. 



f Scale Name . 


Description of Scale 


-Sample Item ; s-j_ 


Moos's Category ! 

It 5gf * . ^ 


Student Support 


There is good rapport between teachers 
and students and students behave in a 
responsible self-disciplined manner. 


There are many disruptive, 
difficult students in the school. (-) 


Relationship 


Affiliation 


Teachers can obtain assistance, advice 
and encouragement and are made to feel 
accepted by colleagues. 


I feel that I could rely on my 
colleagues for assistance if I 
should need it (+) 


Relationship 


Professional 

Interest 


Teachers discuss professional matters, 
show interest in their work and seek 
further professional development 


Teachers frequently discuss 
teaching methods and strategies 
with each other. (+) 


Personal 

Development 


Mission 

Consensus 


Consensus exists within the staff about 
the goals 


Teachers agree on the school’s 
overall goals. (+) 


System Maintenance 
and System Change 


Empowerment 


Teachers are empowered and encouraged 
to be involved in decision making 
processes. 


Decisions about the running of 
this school are usually made by 
the principal or a small group of 
teachers. (-) 


System Maintenance 
and System Change 


Innovation 


The school is in favour of planned 
change and experimentation, and fosters 
classroom openness and 
individualisation. 


Teachers are encouraged to be 
innovative in this school (+) 


System Maintenance 
and System Change 


Resource 

Adequacy 


Support personnel, facilities, finance, 
equipment and resources are suitable 
and adequate. 


The supply of equipment and 
resources is inadequate. (-) 


System Maintenance 
and System Change 


Work Pressure 
Items designated (+) 


The extent to which work pressures 
dominates school environment 

are scored by allocating 5. 4. 3, 2. 1, resoectivelv. foi 


Teachers have to work long hours 
to keep up with the workload. (+) 

r the resDonses Stronplv Affree Acrree Nnt Su 


System Maintenance 
and System Change 



Disagree. Items designated (-) are scored in the reverse manner. Omitted or invalid responses are given a score of 3. 



Fisher, Fraser and Wubbels (1993) have reported validation data for the SLEQ for a number of 
samples including one study of 46 teachers in seven Australian schools. The validation data include 
information about each scale’s internal consistency (Cronbach alpha reliability), discriminant validity 
(mean correlation of a scale with the other seven scales) and the ability of the instrument to differentiate 
between the perceptions of teachers in different schools. The alpha coefficients for different SLEQ 
scales ranged from 0.65 to 0.92 suggesting that each SLEQ scale displays satisfactory internal 
consistency for a scale composed of only seven items. 

The SLEQ consists of 56 items, with each of the eight scales being assessed by seven items. Each item 
is scored on a five-point scale with the responses of Strongly Agree, Agree, Not Sure, Disagree and 
Strongly Disagree. Table 2 describes the nature of the SLEQ by providing a scale description and 
sample item for each scale and shows each scale's classification according to Moos’ scheme. As well. 
Table 2 provides information about the method and direction of scoring of SLEQ items. 

For this study, all of the above mentioned SLEQ scales were used, however construction of the scales 
involved weights which were obtained via Confirmatory Factor Analysis and the method is described 
in a latter section of this paper. 

Assessing the classroom environment 

That classes and schools differ in terms of their learning environments, which in turn influence student 
achievement has been demonstrated by Hattie (1987) who showed that 20% of students in desirable 
climates are better off than students in average classrooms. In the last 25 years there have been 
instruments developed for a range of classroom contexts, such as individualised classrooms (Fraser, 
1990) and constructivist classrooms (Taylor, Dawson & Fraser, 1995). These instruments have been 
employed in a range of studies, with different instruments and scales used in particular studies. 
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Recently, Fraser, Fisher and McRobbie (1996) began the development of a new learning environment 
instrument which incorporates scales that have been shown in previous studies to be significant 
predictors of outcomes (Fraser, 1994) and additional scales to accommodate recent developments and 
concerns in classroom learning, such as equity issues and the promotion of understanding rather than 
rote memorisation. The first version of die new instrument contained the following 9 scales, each 
scale containing 10 items: Student Cohesiveness, Teacher Support, Involvement, 

Autonomy/Independence, Investigation, Task Orientation, Cooperation, Equity and Understanding. 
The new instrument employed the same five-point Likert response scale (Almost Never, Seldom, 
Sometimes, Often, Almost Always) as used in some previous instruments. 

For the purposes of this study, we used 6 of these scales in the student questionnaire, that is, Student 
Cohesiveness, Teacher Support, Involvement, Autonomy/Independence, Task Orientation and 
Cooperation (see Table 3). The construction of the scales involved weights which were obtained via 
Confirmatory Factor Analysis and the method is described in a latter section of this paper. 



Table 3. Description of scales in the CLES and example items 



j SCALE 


Classroom Learning Environment 

Example Item 


Student cohesiveness 


Friendships are made among students in this class. 


Teacher support 


The teacher goes out of his/her way to help students. 


Student involvement 


Students talk with each other about how to solve 
problems. 


Independence 


I have a say in deciding what activities I do. 


Task orientation 


Class assignments are clear so everyone knows what to 
do. 


Cooperation 


Students share their books and other resources with each 
other when doing assignments. 



Student perception of school climate 

Students in this study were asked to respond to items regarding their satisfaction with the school. 
There were 18 items in this measure and three scales. These student satisfaction scales were Teacher 
Support, Friends’ Support and Safety. The construction of the scales involved the use of 
Confirmatory Factor Analysis and the method is described in a latter section of this paper. 



Table 4. Description of items in the student satisfaction scales. 



llfli 

mmMmmmSwe mm 


Student Satisfaction Scale Example Items 


No. Items ^ 


Teacher support 


How well teachers understand my problems. 


7 


Friends' support 


How easy it is to make new friends at my school. 


5 


Safety 


How safe I feel at school. 


6 



Student self-concept 

“That self-concept is related to achievement presupposes that certain 
classroom environments enhance both aspects.” (Hattie, 1992, p. 197). 

In previous research about self-concept, the multidimensional nature has been well documented 
(Byrne, 1984; Hattie, 1992; Marsh, 1990, 1993; Marsh & Shavelson, 1985). The academic 
components of the model have been the focus of attention in relationship to external constructs such as 

best copy available , 
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academic achievement. We included two components of the Marsh Self Description Questionnaire 
(SDQII) designed to measure adolescent self-concepts (Marsh, 1992). 

Included in this study, were two measures of Self-Concept, namely, General Self-Concept and 
Academic Self-Concept each comprised of 10 items. Examples of items from these two measures are 
presented in Table 5. The General Self-Concept scale describes the student’s feelings about 
himself/herself. There are both negative and positive statements related to success and failure in life. 
The Academic Self-Concept scale measures the student’s perceptions about their academic ability and 
potential to be a success at school. The construction of the Self-Concept scales involved the use of 
Confirmatory Factor Analysis and the method is described in a latter section of this paper. 

Table 5. Description of some items from the student Self-Concept scales. 



f " ' " , „ : ' "■ 'y . 

j 

; Scale 


| §|||| 

Example Items 


• 

N6Mtemiia»ailli 


General 


Overall, I have a lot to be proud of. 


10 


Self-Concept 


Overall, I am no good. 

Most things I do, I do well. 

Nothing I do ever seems to turn out right. 
Overall, most things I do turn out well. 




Academic 


People come to me for help in most school 


10 


Self-Concept 


subjects. 

I’m too stupid at school to get into a good 
university. 

If I work really hard I could be one of the best 
students in my school year. 

I get bad marks in most school subjects. 

I learn things quickly in most school subjects. 





Science and mathematics achievement 

For the purposes of this study, a relatively simple multiple-choice test of mathematics and science was 
employed. This test had already been validated internationally for use in the Third International 
Mathematics and Science Study (TIMSS) for 13-14 year old students. The TIMSS tested and 
questioned students, teachers and schools in 200 schools throughout Australia and in 50 other 
countries. The results of the TIMSS are available from the Australian Council for Educational 
Research (Lokan, Ford & Greenwood, 1996) and international findings may be viewed at the World 
Wide Web site Http://wwwcsteep.bc.edu/timss. 

Three different rotated forms of the possible eight tests available were used and the open-ended/free 
response part of the test was not used due to time constraints. There were 18 math em atics test items 
and 18 science test items which had to be completed in 45 minutes. There was reading time and 
example test items provided prior to the commencement of the test. 

Analysis of the test items involved a procedure called Rasch Modelling which scores the test items and 
then estimates the student’s ability on that test item as a function of the difficulty of the test item and 
the student responses to other test items. The result is a score which has a range from approximately - 
3.10 to +4.10. The final science and mathematics achievement measures were constructed using the 
Rasch Model. 
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Table 6. Results for the student Mathematics and Science tests for all schools, by grade. 



Student 


wa 

Mean 


1 Minimum 

■ ’ 


• s - V ft 

V; ^Maxmum^ki 


Mathematics (All) 


1.08 


-3.09 


4.00 


Year 8 Mathematics 


0.80 






Year 9 Mathematics 


1.12 






Year 10 Mathematics 


1.40 






Science (All) 


1.13 


-3.07 


4.10 


Year 8 Science 


0.74 






Year 9 Science 


1.07 






Year 10 Science 


1.74 







THE SAMPLE 

Western Australian schools are located in a variety of locations, which have previously been 
categorized into three groups in other analyses (Tomlinson, 1994; Young, 1994b, 1994d): 
metropolitan Perth, rural and remote. Unfortunately, these three categories did not account for rural 
cities and other types of rural locations (similarly for the remote category). Subsequently, these 
categories have been expanded by the Department of Primary Industries and Energy and the Australian 
Bureau of Statistics (DPIE, 1994) into seven categories, five of which were then used in this study 
(Table 7a). The five categories were Metropolitan (Capital City), Small Rural Centres, Other Rural 
Areas, Remote Centres and Other Remote Areas and these were incorporated into this study. In 
Western Australia, only these five categories are applicable. 



Table 7a. Rural location categories. 



| Classification 


Category ; 


Population Size j 


Metropolitan: 


Capital City 






Other Metropolitan Centre 


urban centre pop > 100,000 


Rural: 


Large Rural Centres 
Small Rural Centres 
Other Rural Areas 


urban centre pop 25,000 - 99,999 
urban centre pop 10,000 - 24,999 
< 10,000 


Remote: 


Remote Centres 
Other Remote Areas 


urban centre pop > 5,000 
<5,000 



Additionally, the sample was stratified further into three categories of Socioeconomic Status (SES). 
Socioeconomic strata was defined using the student SES consisting of mother and father’s occupation 
and education. This SES measure was then aggregated to the school level and categorised into three 
groups of Low, Medium and High average socioeconomic status. For this Western Australian study, 
therefore, the total number of strata were 5 Location strata by 3 SES strata or 15 strata altogether 
(Table 7b). 
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Table 7b. Stratified sample design. 



Strata;Ty^^f; : vV'- • - Strata Categories 'h. f ' '} 



Strata Categories 



Location 



SES 



Metropolitan Perth, Small Rural Centres, Other Rural Areas, 
Remote Centres and Other Remote Areas 

Low, Middle and High Socioeconomic Status 
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Sampling techniques used in this study were developed by Kish (1965) and further refined by Ross 
(1976, 1987). The applicant has had long experience in the use of the stratified complex sample 
design (Young, 1991a). An important feature of this study involves the inclusion of Non-government 
schools. These included Catholic, Anglican and other types of Non-government schools, although no 
stratification was used for these school types. 

There were 3397 students in the achieved sample of students from 28 schools (see Tables 7c and 7d 
below). While the three categories of SES were constructed so as to include approximately equal 
thirds of students, this did not eventuate with the five categories of location. Unfortunately, the 
metropolitan schools sampled were from low and medium SES catchments and this had the effect of 
an empty strata for high SES metropolitan schools. Similarly, high SES schools in remote centres 
were not sampled. In the second stage of this study, 1997, a further sampling will occur and these 
strata will be filled at this time. 

Table 7c. Sample size by rural location. 



Small 

Sample Size Perth Rural 

Centre UK 


gfe Rural 

■/ JSSl.m 


Remote Remote Total 

Centre Area Students 


Students 619 747 

Schools 4 6 


1013 

9 


633 385 3397 

5 4 28 



Table 7d. Sample size by rural location and average socioeconomic status. 



PlipS%mdeht 

Sample Size 
by SES 


Perth 


Rural 


lltmrea. 


Remote 

■■linKISI 


Remote 

• t^AifeaK !•': 


Total 

Students 




Low SES 


276 


355 


152 


98 


156 


1037 


10 


Medium SES 


343 


339 


42 


535 


21 


1280 


10 


High SES 


- 


' 53 


819 


- 


208 


1080 


8 














3397 


28 



Note: While all strata were not filled, this was a problem in terms of sampling by persuasion. It is hoped that the 
smaller cells (strata) will be filled by additional schools being added in 1997. 



METHODOLOGY 

Students from 28 schools were asked to complete a questionnaire, along with a combined mathematics 
and science test. Students in Years 8, 9 and 10 participated, with this cohort to be surveyed and tested 
at least one more time (1997) and preferably two more times (1997 and 1998), depending upon 
availability of the cohort and continuation of funding. The student questionnaire consisted of 
background and socioeconomic questions, along with questions about their rural life. 

In this questionnaire, students completed satisfaction items (3 scales) and self-concept items (2 scales). 
For the Student Satisfaction Scales, the item responses were a five point measure from Very Happy 
(coded 5) to Very Unhappy (coded 1) with the described support. Similarly, the Self-Concept scales 
consisted of a set of statements to which the student responded on a five point measure, from False, 
Mostly False, Neither False nor True, Mostly True to True (coded 1 to 5). Each student was also 
asked about their Classroom Learning Environment and these scales were also estimated for reliability 
(6 scales). A sample of 3397 students was achieved for the 1996 data collection. 

The 106 Science/Mathematics teachers participating from each of the 28 schools completed a Teacher 
Questionnaire, consisting of the School Level Environment Questionnaire (SLEQ), including 8 scales 
and 56 items, and a few other background questions. The SLEQ has already been described 
previously. Teachers mailed their completed questionnaires directly to the research project using a 
reply-paid envelope. 
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Confirmatory factor analysis 

These student and teacher composite scales consisted of items which were categorical, not continuous. 
Additionally, these items varied in their loadings which indicated that Confirmatory Factor Analysis 
was crucial to the effective construction of the composite scale. When the observed variables (items) 
are non-normal and non-continuous, the use of product-moment correlations can lead to large negative 
biases in their estimates (Joreskog, 1990; Carroll, 1961). It is therefore a significant feature of this 
study that Structural Equation Modelling techniques (WLS) were used which assume that the observed 
variables are measured on an interval scale with non-normal distributions. Joreskog (1994, p. 383) 
observed that ordinal variables represent a set of ordered categories, such as the five-category Likert 
scale, which need to be treated differently: 

‘It is common practice to treat scores 1, 2, 3, 4, 5, representing the ordered 
categories of an ordinal variable as numbers on an interval scale and use a 
covariance matrix computed in the usual way to estimate a structural equation 
model. What is so bad with this is not so much that the distribution is non-normal; 
more importantly the distribution is not continuous: there are only four distinct 
values in the distribution.” 

The Weighted Least Squares (WLS) method available in LISREL 8 was developed to assist with the 
analysis of non-normally distributed variables by providing an appropriate weight matrix, correct 
parameter estimates, standard errors and a fit statistic. “The weight matrix required for such an 
analysis is the inverse of the estimated asymptotic covariance matrix W of die polychoric and 
polyserial correlations” (Joreskog & Sorbom, 1993, p. 45). 

In this study, the polychoric correlation matrix and asymptotic variance-covariance matrix were 
produced using Weighted Least Squares (WLS) PRELIS, which was then analysed using LISREL. 
This procedure was used to calculate each composite scale, assuming the one-factor congeneric 
measurement model. 

The one-factor congeneric measurement model (Joreskog, 1971) was used in order to construct a set 
of factor score regression weights using LISREL (Joreskog & Sorbom, 1996). This type of 

measurement model assumes that the error variances (8) and regression coefficients (X.) vary (are non- 
equivalent) as depicted in Figure 2. 

Fitting a congeneric measurement model allows for differences in the contribution each individual 
measure contributes to the overall composite scale (Fleishman & Benson, 1987). In the generalised 

one-factor congeneric measurement model (Figure 2), £ is the latent variable (composite scale); x,, x 2 , 

x 3 , and x 4 are the observed indicator variables; 8,, 8 2 , 8 3 , and 8 4 are the errors associated with the 

measurement of x,, x 2 , x 3 , and x 4 respectively; and A,,, Xj, Xj, and X 4 are the regression coefficients of 

% in the regression of x,, x 2 , x 3 , and x 4 on % respectively (Joreskog, 1971; Holmes-Smith & Rowe, 
1994). 

The estimated composite score ^ for each person is calculated by multiplying each item x s by its factor 
score regression weight. The factor score regression weights are produced by LISREL output when a 
one-factor congeneric measurement model is estimated for a set of items (the output line must include 
the command FS for factor scores to be displayed). These weights (w,, w 2 , w 3 , w 4 ) represent the 

contribution each item x ; makes to the latent variable 
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8 , 

8 , 

8 , 

8 4 




Figure 2. Generalized one-factor congeneric measurement model 



The composite scale construction procedure was then performed as follows: 

£j = Wj* Xj + w 2 *x 2 + w 3 *x 3 + w 4 *x 4 
where w i = (factor score regression weight for item 
sum(factor score regression weights) 

Each weight (w>) was calculated by dividing the factor score regression weight for each item by the 
total sum of the factor score regression weights. This had the advantage of producing a 
proportionately weighted composite scale variable which is adjusted for measurement error (individual 
and joint), is continuous in measurement and ranges from 1 to 5. Missing data were deleted using a 
listwise deletion method. 

Once each scale was computed for each teacher, all teacher results for each school were aggregated to 
the school level (averaged) and presented as a mean. 

Reliability 

That reliability is the consistency of measurement is a concept which has developed from classical test 
theory and assumes that a single true score underlies a measure (Bollen, 1989, p. 221). While 
Cronbach’s (1951) alpha coefficient is the most popular reliability coefficient in social science 
research, it has the weakness of underestimating reliability for congeneric measures. Bollen (1989, p. 
222) recommends using the Coefficient of Determination R 2 xj , as a viable alternative for measuring 
reliability, where structural equations are being used. This is the measure of the proportion of variance 
in a measure which is explained by the variables that directly effect Xj. 

For the purposes of this research, the Coefficient of Determination was used as the measure of 
reliability. The method used was based upon the following formula (Werts, Rock, I, inn & Joreskog, 

1978), where the reliability (r c ) of a composite (i^.) is given as: 

r c = oyd - © 5 )( o c 

ca c '£ co c 
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where CO is a vector of the factor score regression weights, X is the matrix of 
covariances among the observed indicator variables and ©§ is the matrix of 
variances and covariances amongst the measurement error terms 8,, 8 2 , 8 3 , 
and 5 4 



Where the composite scale is obtained by using unit weights, and the vector of weights will be a vector 
of ones and the reliability of the composite becomes (Joreskog & Sorbom, 1996, p. 27): 



r c = 1 - I0 S I 

El 

where l©d is the determinant of the matrix of variances and covariances amongst 

the measurement error terms 8,, 8 2 , 8 3 , and 8 4 and |X| is the determinant of 
the matrix of covariances among the observed indicator variables 

While Cronbach’s Alpha Reliability coefficient is provided in Table 8, the Coefficient of Determination 
is given in order to show the true reliability. All composite scales given in Table 8 were prepared 
using the confirmatory factor analysis described above with factor score regression weights, except for 
Socioeconomic Status which was not weighted due to the different metrics used. Instead, SES was 
calculated with unit weighting and the appropriate reliability coefficient used. 

The achievement test scores were constructed using Rasch modelling procedures and therefore the Infit 
Mean Square is provided as an alternative test of reliability. Further, the achievement test scores were 
kept separate by Year due to the test being different for each year, although of equal ability 
requirement 



RESULTS 

Once the scales were constructed and checked for reliability, they were compared by the five location 
categories and significant differences were found for all scales, except general self-concept (see Table 
9). The Analysis of Variance results showed that there were significant differences between schools 
from the urban and rural locations. 

All scales tended to be lower for students from Remote Centres, however it was suspected that these 
variations may be related to socioeconomic status. Teachers perceived that students were more 
supportive in the metropolitan schools, students perceived that teachers and their own peers were more 
supportive in the country schools. While General Self-Concept of students was equal across the five 
locations, students from remote locations had significantly lower Academic Self-Concept. 

Students in country schools (rural and remote) appeared to be more satisfied with their schools. They 
felt that teachers were more supportive, friends were more supportive and generally felt safer. Science 
and Mathematics Achievement scores were not comparable due to the lack of a prior achievement 
measure in this stage of the study. That is, while there were differences in achievement between 
students from rural and urban locations, the scores were more a reflection of the students’ ability than 
a random selection. 

The numbers of Aboriginal students in this study was higher in the remote centres and areas, leading 
to some confounding of results. Therefore, a further analysis was conducted combining all of these 
possible effects into a single multilevel model of analysis. 
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Table 8. Frequencies, means and reliabilities for student and teacher/school variables. 



•' ’ Variable XSll 




Sampled 

PI 


■Numberjof 

fcSKIIHSI* 


pha> : - v 

Reliability. 


ii^Goemcientl^^S 
: of 

Determination ! 




! Teacher School Level 
l Environment ; 




k ' , 


mm 






' • c<}C'\Jx\4 ■* ' % 


Student Support 




106 


1 


3.60 


.894 


.995 


Affiliation 




106 


1 


4.22 


.871 


.955 


Professional Interest 




106 


1 


3.39 


.765 


.912 


Mission Consensus 




106 


1 


3.64 


.907 


.981 


Empowerment 




106 


1 


3.40 


.715 


.885 


Innovation 




106 


1 


3.55 


.842 


.967 


Resources 




106 


1 


3.57 


.753 


.879 


Work Pressure 




106 


1 


3.91 


.807 


.930 


t Student Classroom Lean 


iingj 






,, . -< --vy -I ,v V"-- 

.x^bir'^S 






i^nvirohment v 






ft t 


• ^ - V ' ; 

■ » ; ■ 




!§ 

;;p? 

\ :: 4 


Student Cohesiveness 




3397 


10 


3.51 


.881 


.910 


Teacher Support 




3397 


10 


2.31 


.919 


.951 


Involvement/ 




3397 


10 


3.26 


.877 


.933 


Negotiation 














Autonomy/Independence 




3397 


10 


4.11 


.691 


.922 


Task Orientation 




3397 


10 


3.45 


.885 


.925 


Cooperation 




3397 


10 


3.40 


.900 


.942 


[^iideht Jel&Coricept Jt^ 












: • ' ' • 1 ' * V.' • ■ . • ' * . 3T! : 


General Self-Concept 




3397 


10 


3.48 


.888 


.944 


Academic Self-Concept 




3397 


10 


3.52 


.889 


.946 


L Student Satisfaction , 




- r. V- . '^CZ 










School Safety 




3397 


6 


3.83 


.783 


.867 


Friends Support 




3397 


5 


3.52 


.785 


.859 


Teacher Support 




3397 


7 


3.05 


.848 


.885 


IMSocioecoriomic^ 


3I1IS 






SSSiffil 2212 


wmsmm 


indent Achievement^^ 








ttS 


mmMMxsiMl 


laaaBi 


Science Achievement 




3397 


18 








Year 8 




1346 




0.00 (.78) 


1.00 (.08) 




Year 9 




1214 




0.00 (.77) 


1.00 (.08) 




Year 10 




1008 




0.00 (.71) 


.99 (.10) 




Maths Achievement 




3397 


18 








Year 8 




1346 




0.00 (.78) 


1.00 (.11) 




Year 9 




1214 




0.00 (.80) 


.99 (.12) 




Year 10 




1008 




0.00 (.76) 


.99 (.10) 
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The Three-Level Multilevel Linear Model: Background 

While there appeared to be differences between rural, remote and metropolitan schools in the initial 
analyses, some of these differences could be due to socioeconomic factors rather than rurality. 
Further, there could be other school or teacher effects which contribute towards explaining these 
differences. Therefore, it is not enough to simply examine location differences and report these 
individually. In order to investigate the influence of location and rurality in explaining differences in 
student achievement, a multilevel linear model of analysis was employed. In this case, a three-level 
model was used where student, class and school comprised the three levels of analysis. 

Traditional linear models on which most researchers have relied upon, require the assumption that 
errors are independent, yet most subjects are ‘nested’ within classrooms, schools, districts, states and 
countries so that responses within groups are group dependent. To ignore the nested structure of this 
type of data ultimately will give rise to problems of aggregation bias (within-group homogeneity) and 
imprecision (Burstein, 1980; Raudenbush, 1988). 

The Multilevel Linear Model provides an integrated strategy for handling problems such as aggregation 
bias in standard error estimates and erroneous probability values in hypothesis testing of school 
effects. For this study, MLn was chosen as the software program appropriate to study school and 
student effects relating to student outcomes. Research on school effects has previously been conducted 
with a set of data analysed at the individual student level, with the assumption that classrooms and 
schools affect students equally. However, when the effects vary among individuals and their 
contexts, this type of statistical analysis can be misleading (Bryk & Raudenbush, 1987). Ordinary 
least squares analysis provides information about the total variance, but can only break this total 
variance into the between- and within-school effects. The between-school effect may be influenced by 
school level variables, such as the affluence of the school. This study endeavoured to explain 
variations in student outcomes by first decomposing observed relationships into between- and within- 
school components. 

Previous studies have shown clearly that educational researchers need to account for the inherent 
multilevel structure of data collected from schools and this literature includes Mason et al. (1983), 
Bosker and Scheerens (1989), Bryk and Raudenbush (1986, 1989, 1992) and Goldstein (1984, 1987, 
1995). 
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Table 9. Means and ANOVA F-test for student and teacher/school variables by location and for total 
sample. 



. Variable 




r^S^iatl ■ ; 

-'S^^Rurai', • 


•; • <. 

;m Other' 
Rural Are 


j|gr7\ ifif 
Sf't.S'Remote'S 

ace,; Centre: , • 


Other 

• v.^ 

a: Remote 

Aiea ?: r s: 


■ ^yTotal 


ANOVA I 

•^S.' ; ;F-test ;>||!| 

(Sig) 


















Teacher SchoollLevel . 5\. 


'St, v: - 




■ •' a 




■ ■. 


isiS.;,. 




Student Support 


3.32 


3.91 


3.84 


2.85 


3.97 


3.60 


9.63** 


Affiliation 


4.36 


4.40 


4.08 


4.16 


4.12 


4.22 


1.30 


Professional Interest 


3.63 


3.70 


3.49 


3.64 


3.91 


3.39 


0.99 


Mission Consensus 


3.27 


3.74 


3.30 


3.09 


3.59 


3.64 


2.53* 


Empowerment 


3.38 


3.56 


3.41 


3.16 


3.50 


3.40 


0.88 


Innovation 


3.36. 


3.79 


3.38 


3.57 


3.78 


3.55 


1.37 


Resources 


3.68 


3.69 


3.45 


3.32 


3.92 


3.57 


1.61 


Work Pressure 


3.89 


4.01 


3.87 


4.02 


3.64 


3.91 


0.54 


pSte'dem Classroom 










■ sv'.s 






^iShiiig' ^yironment ; 






mm y W- 


’% '' ^ 


it 


<>■ jnK!;# . 




Student Cohesiveness 


3.39 


3.52 


3.58 


3.42 


3.57 


3.51 


10.00** 


Teacher Support 


2.19 


2.30 


2.37 


2.32 


2.42 


2.31 


8.33** 


Involvement/Negotiation 


3.20 


3.23 


3.29 


3.20 


3.44 


3.26 


7.79** 


Autonomy/Independence 


4.13 


4.13 


4.11 


4.04 


4.06 


4.11 


1.60 


Task Orientation 


3.33 


3.53 


3.51 


3.32 


3.47 


3.45 


12.42** 


Cooperation 


3.33 


3.39 


3.45 


3.35 


3.48 


3.40 


4.01** 


s Student Self-Concept • 


■ -O' ' 


o ■ ‘ • 


• .'SSx . . .. ■ . 


-11§S ! 


... l . .<? 

S a 'T : S^s'M r 


' / 7 




General Self-Concept 


3.45 


3.51 


3.49 


3.36 


3.64 


3.48 


10.02** 


Academic Self-Concept 


3.40 


3.54 


3.58 


3.44 


3.60 


3.52 


10.66** 












& 


.Sy: 

hi 




School Safety 


3.87 


3.86 


3.84 


3.76 


3.64 


3.83 


5.57** 


Friends Support 


3.37 


3.61 


3.51 


3.41 


3.72 


3.52 


16.69** 


Teacher Support 


2.94 


3.07 


3.14 


2.91 


3.18 


3.05 


16.72** 


[ Student Achievement ’’ 


■ > ' - 'CSs': 


.’V ; 


T!/ '/ • ' ,r - - ' - 
- 


•* ' * /■ \ l \ ^ 


•y . >aS'- ; ;r ll-a 






Science Achievement 


1.21 


1.15 


1.29 


6.93 


0.82 


% V ■ ■ 

1.12 


15.64** 


Maths Achievement 


1.24 


1.11 


1.28 


0.75 


0.67 


1.07 


27.10** 



3.«S' : . 412'-: ’it ••3.95 

1 ' " ““ 



[ Student Numbers; ' 619 . 747 - i 1013 . \ r- 633 

Note: * indicates that the F test was significant at D < 0.05 level. ~ “ 



indicates that the F test was significant at p < 0.05 level, 
indicates that the F test was significant at p < 0.01 level 
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The Three-Level Multilevel Linear Model: The Variables 

The response variables for this analysis were Science and Mathematics Achievement (two 18 item 
scales). There were six different types of variables used in the multilevel analysis (shown below). 
While some analyses described earlier suggested that rural schools may be disadvantaged, the findings 
were unclear. The multilevel analyses combined all of the possible explanatory variables under 
investigation here and revealed how they combine to influence student attitudes. 



Science Achievement 


A student science achievement test 
consisting of 18 multiple choice response 
format test items selected from the Third 
International Mathematics and Science 
Study (TIMSS) and total score estimated 
using the Rasch Model. 


Math Achievement 


A student math achievement test 
consisting of 18 multiple choice response 
format test items selected from the Third 
International Mathematics and Science 
Study (TIMSS) and total score estimated 
using die Rasch Model. 


Student Satisfaction 


A student measure described previously 
as the student’s attitude to and satisfaction 
with the school (continuous and 
standardized). 


Location 


A five category measure described 
previously: Metropolitan Perth, Small 
Rural Centre, Other Rural Areas, Remote 
Centre and Other Remote Areas (1 to 5). 


SES 


Socioeconomic Status of the students 
consisting of mother and father’s 
occupations and education (continuous 
and standardized). 


Average SES 


Socioeconomic Status of the students in 
the school aggregated to the school level 
(continuous and standardized). 


SLEQ Scales 


Eight measures of the teachers’ 
perceptions of the school environment 
aggregated to the school level (continuous 
and standardized). 


CLE Scales 


Six measures of the students’ perceptions 
of the classroom learning environment 
kept at the student level (continuous and 
standardized). 


Self-Concept 


Two measures of the students’ self- 
concept: Academic Self-Concept and 
General Self-Concept (continuous and 
standardized). 
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The Three-Level Multilevel Linear Model: Unconditional Statistical Model 

In this study, the use of the multilevel linear model involved the single cross-section of data with a 
three-level structure consisting of students (Level 1) nested within classes (Level 2) nested within 
schools (Level 3). 

The simplest model was used first, that is, the fully unconditional model with no predictor variables 
specified. The outcome measures, science and mathematics achievement, were free to vary across 
three different levels of analysis: student, class and school. This model is described below in 
Equations 1, 2 and 3. 

Student-Level Model Science/Mathematics Achievement for each student was estimated as a 
function of the class average plus random error: 



where 
Aeh ij k 



Achyj. - Ttq/jfc + e ijk Equation 1 

represents the Science/Mathematics Achievement of each student i in classy and 
school k. 



n 0jk represents the class mean Science/Mathematics achievement of classy in school k. 
Zijk represents the random error of student i in class j and school k 
i = 1, 2, 3, . . ., n ijk students in class j and school k. 
j = 1,2, ...J t classes within school k, 
k= 1, .... K schools. 



Class-Level Model Science/Mathematics achievement classroom mean varies as a function of the 
school mean plus random error: 



n 0jk ~ Poo* + T 0jk Equation 2 

where 

Poo* represents the mean Science/Mathematics achievement in school k. 

Toy* represents the random error of class y within school k 

School-Level Model Science/Mathematics school mean achievement varies randomly around a 
grand mean for all schools. 



where 

7000 

H00* 



Poo* - 7000+ MOO* Equation 3 

represents the grand mean Science/Mathematics achievement for all schools. 

represents the random school effect, the deviation of school k’s mean from the 
grand mean. 



This three-level model partitions the total variability in the outcome measure, 

Science/Mathematics achievement, into its three components: students within classes (a 2 ), 
classes within schools (r n ) and between schools (tp). 

The Three-Level Multilevel Linear Model: Contextual/Student Background Statistical Model 

In order to investigate the effect of the student background and context variables upon student 
achievement in science and mathematics, this model was estimated first using Average Socioeconomic 
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Status (AvSES) of students in school j. Location of school j. Socioeconomic Status (SES) of student 
i. Sex of student i, Aboriginality (Ab) of student i, and English speaking background (Eng) of student 
i. For the purposes of this model, the intercept was allowed to vary across classes and schools. That 
is, mean achievement varied between classes due to classroom effects and schools due to school 
effects. This model is described in Equation 4 below and results presented in the next section (Tables 
10a and 10b). Achy* represents Science and Mathematics achievement which were estimated in 
separate models. 

In the equation presented below, Ach^ is the Science/Mathematics achievement of student i in class j 
and in school k. This is the student level equation. There is one random equation and six fixed 
effects equations presented next, with the mean achievement UQj k allowed to vary between classes. 
This is the classroom level equation. Finally, there is one random equation at the school level, where 
the grand mean achievement Poo* is allowed to vary across schools. This is the school level equation. 
Together these separate equations make up the statistical model used to estimate the effects of context 
and student background variables on student achievement. Two separate analyses were conducted for 
science and mathematics achievement. 

Ach ijk = n 0jk + 7 i l 7 *(AvSESyfc) + ^(Locationj/jfc) + n 3jk (SESij0 + 



The Three-Level Multilevel Linear Model: Four Conditional Statistical Models 

Upon estimation of the contextual/student background model, four further conditional models were 
estimated in order to investigate the effects of the school environment, student satisfaction, classroom 
learning environment and student self-concept on achievement. For each of these four models, the 
contextual effects and home background were controlled for. Additionally, these models were 
estimated separately for science and mathematics achievement. The four generic statistical equations 
are presented below (Equations 5 to 8 ). 

For the first model estimating the effect of the School Level Environment Questionnaire on student 
achievement, eight school level variables were included (Equation 5). These were teacher variables 
which were aggregated to the school level. For the purpose of the statistical model, they really were 

ttjjk to 7 i] 4 ^ as the eight variables were estimated separately. These were Student Support, 
Affiliation, Mission, Professional Interest, Empowerment, Innovation, Resources and Work Pressure 
(described previously). 

For the second model estimating the effect of Student Satisfaction on student achievement, three 
student level variables were included (Equation 6 ). These were Teacher Satisfaction, Friends and 
Safety, as described previously. 



n 4/*(S exyfc) + 7t5 7 *(Abyfc) + n 6 j k (Engij0 + ep 




Poo* = T000/+ W)0* 



Equation 4 
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Achy£ — 



™Ojk + ^l/iCAvSESy^) + n2jk(Locationij0 + Ky k (SESij0 + 
Ti 4jk (Sexij0 + n 5jk (Abij0 + ^6jk^Bijk) + n ljk (SLEQ p) + ep 
n Ojk = PoO* + r 0jk *4jk = P4OO 

n ljk = PlOO ™5jk = P 500 

n 2jk = P2OO n 6jk = P600 

% 3jk = P 300 n 7 jk = p 700 

Poo* = Y000/+ MOO* Equations 



A ch ijk = 



n 0jk + ^i^(AvSESy^) + ^(Locationyjt) + ic^SESgft) + 
n 4jk (Sexij0 + n 5jk (Abij0 + K 6Jk (Eng i j0 + 7i 7y *(TeachSaty*) 
7t 8 ^(FriendSyjt) + ^^(Safety,^) + ep 



= Poo* + r q/* 
= P100 
n 2jk = P200 
= P300 

Tty* = P 400 

POO* = Y000/+ MOO* 



™5jk 

n 6jk 

n ljk 

n Sjk 

n 9jk 



Psoo 

P 600 

P 700 

P 800 

P 900 



Equation 6 



Achy* = n 0jk + 7ti 7 *(AvSESy*) + ^(Locationy*) + % ijk { SESyjfc) + 



^4y* ( ^ CX y *) + ^ 5 y*(Aby*) + }[$j^(Eng|jJ^ + ^ 7 /*(CLEy*) + Cy* 



*q/* = Poo* + r o/* 

71 17* = P100 

n 2jk ~ P200 

K 3jk = P 300 



w 4/* = P4OO 

^57* = Psoo 
^j* = P600 
^77* = P700 



Poo* = Y 000 /+ Moo* 



Equation 7 



Achy* = 



*07* + ^l7*(AvSESy*) + 7t2 / *(Locationy*) + ^(SESy*) + 
7t 4j *(SeXy) + 7t 5j *(Aby*) + n 6jk (Engij k ) + ^(AcSelfConp) + 
7t 8j *(GenSelfCony*) + ep 

^07* = Poo* + r 07* ^47* = P400 

71 17* = PlOO ^57* = Psoo 

^27* = P200 ^67* = P 600 

^37* = P 300 ^77* = P 700 



POO* = Y000/+ MOO* 



^j* = P 800 



Equation 8 
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For the third model, the effect of six Classroom Learning Environment scales were estimated as 
described previously, and shown in Equation 7. These were student level variables and included 
Cohesion, Teacher Support, Involvement, Independence, Task Orientation and Cooperation. 

The fourth model estimated the effect of two student level variables: Academic Self-Concept and 
General Self-Concept. These were described previously and the statistical model presented in 
Equation 8. 



DISCUSSION 



The Three-Level Multilevel Linear Model: Student Achievement and Contextual/Background Effects 

Firstly, the variation in student science and mathematics achievement was decomposed at the three 
levels as shown in Tables 10a and 10b. Most of the variation in science achievement was found to be 
at the student level (63.9%), with 34.4% variation between classes and 1.7% variation between 
schools (Table 10a). Similarly, the variation in mathematics achievement was mostly between 
students, with 59.3% of the total variance between students, 34.6% between classes and 6.0% 
between schools. These variance components are further illustrated in Figures 3 and 4 for Science 
Achievement and Mathematics Achievement respectively. In these figures, the variance between 
schools, classes and students is represented as a pie diagram. 



Table 10a. Variance components for three-level multilevel model for science achievement. 




Fixed Model 


Constant 


1 .078 (0.070) 




RandomiModel ; 


Hi ;t?Pairameter ;iW^p^:'^>Variance Estimate- 


:;SK|Kereentage of Tdti 


! 

p| 


School 


Constant 


0.025 (0.038) 


1.7 % 




Class 


Constant 


0.531 (0.074) 


34.4 % 




Student 


Constant 


0.985 (0.025) 


63.9 % 




Total 




1.541 


100.0 % 




Table 10b. Variance components for three-level multilevel model for mathematics achievement. 
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Fixed Model 


Constant 


0.988 (0.095) 
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School 


Constant 


0.113 (0.068) 


6.0 % 




Class 


Constant 


0.655 (0.091) 


34.6 % 




Student 


Constant 


1.123 (0.029) 


59.4 % 




Total 




1.891 


100.0 % 
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Variance Percentage 
Science Achievement 




■ School 

■ Class 

■ Student 



Figure 3. Pie Diagram Depicting Variance Components for Science Achievement 



Variance Percentage 
Math Achievement 



\ 




*■ School 
■ Class 
* Student 



Figure 4. Pie Diagram Depicting Variance Components for Mathematics Achievement 
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Second, two school level variables were estimated with a reduction in school level variance (see Table 
11). The effects were positive for school average SES, that is achievement was higher for those 
students attending schools where their peers came from higher socioeconomic backgrounds. The 
effect was also strong and positive for school location, that is achievement was higher for those 
students attending rural and remote schools, even when average SES was accounted for. It was 
interesting to note that these two school variables differentiated between classes for both science and 
mathematics achievement They were also statistically significant for science and mathematics 
achievement with the estimated effects being greater than 2 standard errors. 

Third, four student level variables were included in this three level model: student socioeconomic 
status (SES), student sex (Sex), Aboriginality (Ab) and English Speaking Background (Eng). The 
SES effect was weak and positive (and significant being greater than two standard errors), while the 
gender effect was strong and positive (significant). That is, boys were outperforming girls in both 
mathematics and science. While the effect of being Aboriginal or Torres Strait Islander was strong and 
negative, the effect of speaking English in the home most of the time was positive on achievement 
These effects, were both strong and accounted for a significant proportion of variation in student 
achievement. The effect of School mean SES, Location, SES, Gender, Aboriginality and English 
Speaking Background on science achievement explained 10.0 % of the total residual variance for 
science achievement and 1 1.2 % for mathematics achievement (Table 11). 

Overall, these student background and contextual effects accounted for a significant proportion of 
unexplained variance in science and mathematics achievement. It is clear that they cannot be ignored 
when estimating the effects of other organisational or teacher factors. 

The Three-Level Multilevel Linear Model: Student Achievement and the School Level Environment 
Questionnaire (SLEQ) 

The teachers participating in this study completed a School Level Environment Questionnaire (SLEQ) 
in order to measure their perception of this school’s work environment. These results were aggregated 
to the school level and the effect on achievement estimated with the three-level multilevel linear model. 
There were no significant effects upon student achievement noted. Further, the SLEQ variables did not 
explain residual variance in student achievement. These results are depicted in Table 12 for Science 
and Mathematics Achievement 

The Three-Level Multilevel Linear Model: Student Achievement and Student Satisfaction 

Students were asked to report on their satisfaction with the school in terms of teachers, friends and 
their perception of safety at the school. Effects upon achievement were small, with little of the 
variation in achievement explained either in science or mathematics (Table 13). Student satisfaction 
with their teachers was the most positive and significant effect on achievement 

The Three-Level Multilevel Linear Model: Student Achievement and the Classroom Learning 
Environment 

Of the six Classroom Learning Environment scales, there were few significant effects. The 
supportiveness of the teacher was positively related to achievement while independent/autonomy 
learning in the classroom was negatively related to achievement For both science and mathematics 
achievement little of the residual variance was explained by the Classroom Learning Environment: 
3.6 % for science achievement and 3.0 % for mathematics achievement (see Table 14). 

The Three-Level Multilevel Linear Model: Student Achievement and Student Self-Concept 

There were two Self-Concept measures estimated in this study. Academic Self-Concept had a strong 
and positive effect upon both science and mathematics achievement (Table 15). That is, students who 
perceived that they were capable and confident of their academic ability, tended to outperform their 
peers in both science and mathematics achievement. While the effect of general self-concept was also 
estimated, it had a weak effect upon achievement which was not explainable. Overall, Self-Concept 
accounted for 6.3 % of the residual variance in science achievement and 9.8 % in mathematics 
achievement. 
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Table 1 1 . Three-level multilevel analysis of student SES, school mean SES, location, gender. 
Aboriginally and English-speaking background for science and mathematics 
achievement. 




Fixed Model 




0.998 (0.173) 
0.202 (0.058) 
-0.144 (0.047) 
0.071 (0.018) 
0.224 (0.035) 
-0.526 (0.066) 



English Spk Bk. 

P ; • PI: 



1.273 (0.196) 
0.223 (0.067) 
-0.200 (0.054) 
0.093 (0.020) 
0.145 (0.038) 
-0.584 (0.071) 
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School 


Constant 


0 


1.6% 


0.001 (0.035) 


5.9 % 


Class 


Constant 


0.445 (0.057) 


5.6% 


0.593 (0.083) 


3.3 % 


Student 


Constant 


0.942 (0.024) 


2.8 % 


1.085 (0.028) 


2.0 % 


Total 




1.387 


10.0 % 


1.679 


11.2% 
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Table 12. Three-level multilevel analysis of school level environment for science and 

mathematics achievement 




Fixed Model 



SLEQ 



Constant 

Average SES 

Location 

SES 

Gender 

Aboriginal 

English Spk Bk. 

Student Support 

Affiliation 

Mission 

Professional Int 

Empowerment 
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1.083 (0.182) 
0.256 (0.077) 
-0.177 (0.052) 
0.071 (0.018) 
0.224 (0.352) 
-0.530 (0.066) 
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-0.038 (0.086) 
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-0^084 (0,058) 
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0 


0% 


0 
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1.7% 
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0% 
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0% 


Total 
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0.9% 
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1.7 % 
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Table 13. Three-level multilevel analysis of student satisfaction with school for science and 
mathematics achievement. 




Fixed Model 



Student Sat 



Constant 

Average SES 

Location 

SES 

Gender 

Aboriginal 

English Spk Bk. 

Teacher 

Satisfaction 

Friends 



Safety ^ 



' j 

School 
Class 
Student 



Constant 

Constant 

Constant 



1.007 (0.173) 
0.204 (0.058) 
-0.148 (0.047) 
0.067 (0.018) 
0.223 (0.035) 
-0.517 (0.065) 
0.480 (0.089) 
0.093 (0.020) 

-0.052 (0.021) 
-0.005 (0.022) 
Variance Estimate 
0 

0.446 (0.057) 
0.934 (0.024) 
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1.281 (0.195) 
0.224 (0.066) 
-0.203 (0.054) 
0.089 (0.020) 
0.143 (0.038) 
-0.577 (0.070) 
0.316 (0.095) 
0.076 (0.022) 



0% 

0.5 % 



-0.049 (0.023) 
0.010 (0.024) 
Variance Estimate 
0 

0.587 (0.074) 
1.080 (0.027) 




0% 
0.3 % 
0.3 % 



Total 



1.380 



0.5 % 



1.667 



0.6 % 
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Table 14. Three-level multilevel analysis of the classroom learning environment for science and 
mathematics achievement 
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1.331 


3.6 % 


1.625 
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Table 15. Three-level multilevel analysis of student self-concept on science and 
mathematics achievement 
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0% 


Class 


Constant 


0.395 (0.051) 


3.2% 


0.483 (0.068) 


5.8 % 
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Constant 


0.894 (0.023) 


3.1 % 


1.010 (0.026) 


4.0 % 


Total 




1.289 


6.3 % 


1.499 


9.8 % 



The importance of the positive effect of Academic Self-Concept upon student performance was noted 
to vary across classrooms, as well as between students. That is, students who resided in classrooms 
where their peers had strong Academic Self-Concept tended to outperform students who resided in 
classrooms where their peers had weaker perceptions of themselves. This classroom effect accounted 
for 3.2 % of the residual variance in science achievement and 5.8 % of the residual variance in 
mathematics achievement. This effect was interesting in that it was more than half of the student level 
variance explained. 



DISCUSSION 

In these tables (Tables 11 to 15) a number of different school and classroom effects were measured for 
their effect upon student achievement, while accounting for student background, school location and 
average SES of the school. While these background/context variables were statistically significant and 
explained a large amount of the variance in achievement, little was found to influence student 
achievement at the school or classroom level of analysis. To start with, the school effect was 
negligible (1.7 % for science achievement and 6.0 % for mathematics achievement). Teacher scales 
did little to explain student achievement, nor did student satisfaction with the school. The student level 
effects of the classroom learning environment and student self-concept were more influential, with 3.6 
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% and 3.0 % of the variance explained by the classroom learning environment and 6.3 % and 9.8 % 
explained by student self-concept (for science and mathematics achievement respectively). 

DEFINING AN EFFECTIVE SCHOOL 

In this paper, it has been assumed that an effective school will produce students of higher 
achievement, a greater sense of satisfaction with the school, friends and teachers, and a more positive 
perception of their classroom. Further to this, the teachers will feel more satisfied with their leadership 
and their workplace. 

However, there is a more important aspect which some researchers purport that defines an effective 
school. This v is value adding. For a school to be truly effective in producing students of higher 
performance than when they arrived at the school, the school must consider itself as an organisation 
designed and purposed to add value to their cliental: the student. While the happiness of the teachers, 
parents and leadership is a considerable asset to the improved functioning of the school, the 
improvement of the student must be its primary goal. 

How can a school add value and how can this be measured? Schools can add value to the students’ 
self-esteem (self-concept), attitude and achievement (Hill, 1996). Of course, mathematics and science 
achievement are only two measures of achievement. Others may include reading, English, numeracy 
etc... Schagen defines the term “Value-Added” as: 

The assessment of pupil performance taking into account the prior 
attainment of the pupils concerned. It is largely synonymous with 
“Progress”, the increase in some form of educational attainment made by 
an individual pupil during a period of schooling. (Schagen, 1996) 

So how can the research deal with these issues in a manageable methodology? 

There are two approaches which may be taken. Student ability could be measured using an ability test 
or intelligence test. However, these are only practical if the tests are paper and pencil. Credible and 
reliable ability tests are usually oral and involve a trained psychologist who works with the student on 
an IQ test one to one. This is unreasonably expensive when there are a large number of students 
involved. If ability was successfully measured, it would usefully control for student ability when 
investigating the effects of other variables on achievement (e.g., location). It would also be most 
useful for successfully controlling for uneven sampling of intact classes within schools, where the 
intact classes are streamed into ability groups (as happened with this study). 

A second approach is the longitudinal method. By measuring student cognitive ability (achievement) 
and attitudes over more than one time point, the effectiveness of the school in adding to the students’ 
performance and attitudes can be more adequately measured. This procedure is more expensive and 
time consuming for the researcher, but provides a useful way to account for students’ prior ability in a 
subject area and their attitudes as well. 

This research study intends to follow the longitudinal approach and collect data from this cohort of 
students over three time points. In 1997, the same cohort of students will be tested and surveyed, 
along with an addition to the cohort of some other schools which will fill out the sampling frame 
(Table 1). These additional schools will probably be tested only for two time points, although this 
decision will probably depend upon available funding and time. It is expected that die quantitative 
stage of this study will finish in 1998, although some case study work may still continue into 1999. 

CASE STUDIES OF EFFECTIVE SCHOOLS 

Although funding has not been provided to cover adequate case study analyses of the outiier schools, 
this is considered to be an important component to the overall strategy of the Western Australian 
School Effectiveness Study. Therefore, case studies will commence in 1997 and funding will be 
applied for. Case studies are expected to take two to three years to complete and hopefully form a 
complete study in their own right 

Outlier schools are those schools which show particular interest in terms of their effect on students, 
teachers and parents. Some variables not mentioned previously will be examined, such as teacher 
satisfaction with the leadership and parental satisfaction with the school, and a few schools selected for 
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intensive case study work. While many schools have already volunteered to participate in this aspect 
of the study, careful checking of the data will be done first and then selections made on this basis. 

SCHOOL FEEDBACK 

An important feature of this study is the school feedback. Each school has been provided with a report 
for each of the scales described in this study with a description of the scale and a comparison with the 
mean score for all schools participating. 

Many principals, deputy principals and teachers have commented on the usefulness of this report, 
especially as a platform for staff discussion. It has also helped me to identify some inadequacies of 
this research such as the limited teacher sample from some schools. In every case, the schools were 
satisfied that the findings were constructive to their own understanding of their school and provided 
the school with the opportunity to improve some of the factors which could help with school 
development. 



CONCLUSION 

For this type of response variable, most of the variation in both science and mathematics achievement 
was at the student level. While the rural location and the socioeconomic status of the student’s school 
were an important part of this study, and there were class and school effects, the student’s self-concept 
was the most influential variable investigated here. For students from country schools and city 
schools, the important feature here was their own self-concept. How they believed in their own 
ability, their self-esteem, their belief that they can and will do well - these were the factors which 
influenced their achievement It was also an important finding that the student’s classroom peers’ self- 
concept affected that student’s achievement Perhaps self-concept has a mass effect within the 
classroom. Further, the supportiveness of the teacher and the students’ satisfaction with the teachers 
positively influenced the science and mathematics outcomes. 

This paper incorporated a comparison of five rural locations. Because the results are based upon 
streamed classes of both high and low ability students, how the variables interacted were unclear, so 
the multilevel linear model was used to combine all of the variables under investigation. By building 
the model in a step-wise fashion, the amount of variance explained by each set of variables was 
determined. This is a very useful statistical tool, which is not exploratory, but rather confirmatory in 
approach. That is, we set out to test the hypothesis that these variables were explanatory of student 
achievement in science and mathematics. 

There is still a great deal of student variance left unexplained by this model. It is therefore important to 
continue to test further hypotheses in order to determine what makes a student happy and content with 
the school and what can improve student achievement. 
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